Evaluating classifier performance with highly imbalanced Big Data

Abstract Using the wrong metrics to gauge classification of highly imbalanced Big Data may hide important information in experimental results. However, we find that analysis for performance evaluation and what they can or reveal is rarely covered related works. Therefore, address gap by analyzing multiple popular on three tasks. To best our knowledge, are first utilize new Medicare insurance claims datasets which became publicly available 2021. These all imbalanced. Furthermore, comprised completely different data. We evaluate five ensemble learners Machine Learning task fraud detection. Random Undersampling (RUS) applied induce class ratios. The classifiers evaluated with both Area Under Receiver Operating Characteristic Curve (AUC), Precision Recall (AUPRC) metrics. show AUPRC provides a better insight into performance. Our findings AUC metric hides impact RUS. results terms RUS has detrimental effect. that, Data, fails capture about precision scores false positive counts reveals. contribution more effective evaluating when working Data.